Hippocampal Formation Breaks Combinatorial Explosion for Reinforcement Learning: A Conjecture

There is surmounting evidence that reinforcement learning (RL) is a good model for the dopamine system of the brain and the prefrontal cortex. RL is also promising from the algorithmic point of view, because recent factored RL algorithms have favorable convergence and scaling properties and can counteract the curse of dimensionality problem, the major obstacle of practical applications of RL methods. Learning in navigation tasks then separates (i) to the search and the encoding of the factors, such as position, direction, and speed, and (ii) to the optimization of RL decision making by using these factors. We conjecture that the main task of the hippocampal formation is to separate factors and encode into neocortical areas the different lowdimensional conjunctive representations of them to suit factored RL value estimation. The mathematical framework is sketched. It includes convergent factored RL model and autoregressive (AR) hidden process model that finds factors including the hidden causes. The AR model is mapped to the hippocampal formation.